GAO 

Accountability  *  Integrity  *  Reliability 


An  Assessment  of  Acquisition  Outcomes 
and  Impact  of  Reforms  &  Initiatives  --  2011 


Presentation  by 
Michael  J.  Sullivan 

Director,  Acquisition  &  Sourcing  Management  Team 
U.S.  Government  Accountability  Office 

Naval  Post  Graduate  School 
9th  Annual  Acquisition  Research  Symposium 

Monterey,  CA 
May  17,  2012 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

17  MAY  2012  2- REPORT  TYPE 

3.  DATES  COVERED 

00-00-2012  to  00-00-2012 

4.  TITLE  AND  SUBTITLE 

An  Assessment  of  Acquisition  Outcomes  and  Impact  of  Reforms  & 
Initiatives  --  2011 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Government  Accountability  Office, 441  G  street 

NW, Washington, DC, 20548 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  OS 

unclassified  unclassified  unclassified  Report  (SAR) 

24 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


GAO 

Accountability  *  Integrity  *  Reliability 


2012  Assessment  Made  Observations 
On  The  Following 


•  Cost  performance  and  characteristics  of  the 
Major  Defense  Acquisition  Program  portfolio 

•  Timing  and  amount  of  knowledge  achieved 


Progress  in  implementing  acquisition 
reforms  and  department  initiatives 
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Cost  Performance  and  Characteristics  of 
DOD’s  Portfolio  of  Major  Defense 
Acquisition  Programs 
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Obsevations  about  portfolio’s  cost 


•  Estimated  cost  of  201 1  MDAP  Portfolio  is  $1 .58T  and  has  grown  by  $74B, 
or  5%,  in  past  year 

•  About  $30B  resulting  from  quantity  changes,  $45B  due  to  RDTE  and 
production  inefficiency 

•  Programs  with  greatest  RDTE  growth  are  in  production 

•  The  F-35  program  accounts  for  21%  of  the  portfolio’s  total  cost  and  52%,  or 
about  $39B,  of  its  cost  growth  in  the  past  year 

•  91%  of  funding  needed  to  complete  programs  in  the  portfolio  is  for 
procurement,  most  of  which  is  for  a  few  large  programs 

•  60%  of  the  96  programs  in  the  MDAP  have  lost  buying  power  over  the  past 
year,  depriving  DOD  of  funding  for  other  priorities 

•  About  40%  of  MDAPs  exceeded  cost  growth  targets  in  past  year 

•  The  number  of  MDAPs  is  smaller  this  year  than  last  and  projected  to  be 
smaller  next  year 
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1yr/5yr/Baseline  Trend:  FY  2011  MDAP 
Portfolio  Cost  Growth  Over  Time 


Fiscal  year  2012  dollars 
in  billion 

1  -year 
comparison 
(2010  to  2011) 

5-year 
comparison 
(2006  to  2011) 

Since  first  full 
estimate 
(baseline  to 
2011) 

Increase  in  total  research 
and  development  cost 

$14  billion 

4  percent 

$39  billion 
14  percent 

$113  billion 
54  percent 

Increase  in  total 
procurement  cost 

$61  billion 

5  percent 

$192  billion 

1 9  percent 

$321  billion 
36  percent 

Increase  in  total 
acquisition  cost 

$74  billion 

5  percent 

$233  billion 

1 7  percent 

$447  billion 
40  percent 

Average  delay  in 
delivering  initial 
capabilities 

1  month 

2  percent 

9  months 
11  percent 

23  months 
32  percent 

Source:  GAO  analysis  of  December  2010  Selected  Acquisition  Reports,  prior  Selected  Acquisition  Reports,  and  other 
DOD  data. 
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Performance  of  DOD’s  2011  Portfolio  of 
MDAPs  over  the  Past  Year  (cont.) 


•  RDT&E  costs  increased  $14  billion  from  2010  to  201 1 .  JSF, 
Chemical  Demilitarization — Assembled  Chemical  Weapons 
Alternatives,  SBIRS  High,  F-22  Raptor,  and  P-8A  Poseidon  had  the 
largest  increases,  totaling  $8.3  billion.3 

•  Procurement  costs  increased  $60.6  billion  from  201 0  to  201 1 ,  of 
which  $29.6  billion  can  be  attributed  to  quantity  changes. 

•  The  Littoral  Combat  Ship,  Joint  Mine  Resistant  Ambush  Protected 
vehicle,  DDG  51  destroyer,  HC/MC-130,  and  F/A-18  E/F  programs 
experienced  the  largest  cost  increases  due  to  increased  quantities 
and  account  for  $52  billion  in  growth.  This  is  partially  offset  by  large 
reductions  on  MEADS  and  EFV. 

•  Procurement  costs  for  JSF  increased  by  $34.7  billion  over  the  last 
year  without  any  changes  to  its  quantities. 


a  The  NPOESS  SAR  showed  an  increase  of  $0.9  billion,  but  the  program  has  been  cancelled. 
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Joint  Strike  Fighter  Drives  Much  of 
Portfolio’s  Remaining  Funding  Needs 


Total  sunk  cost  through  2011 


Funding  needed  to  complete 


Top  20  programs 

Joint  Strike  Fighter 

Virginia-class 

Submarine 

Littoral  Combat  Ship 
P-8A  Poseidon 
CH-53K  Helicopter 
CVN  78  Class 
Black  Hawk  Helicopter 
V-22  Osprey 
DDG  51  Destroyer 
JTRS  GMR 


$256.0  billion 


Procurement  through  fiscal  year  2011 

Research  and  development  through  fiscal  year  2011 

Procurement  fiscal  year  2012  to  completion 

Research  and  development  fiscal  year  2012  to  completion 


Source:  GAO  analysis  of  DOD  data. 
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Joint  Strike  Fighter  Accounts  for  Significant 
Portions  of  the  Portfolio’s  Growth 


Joint  Strike  Fighter  as  a  Portion  of  201 1  Portfolio  Cost  Growth 


$3.9  billion 

■■■■■■■■■■ 

■■■■■■■■■■ 

■■■■■■■■■■ 

$34.7  billion 

$38.6  billion 

■■■■■■■■■■ 
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Research  and  development 

Procurement 

Total  acquisition  cost 

Unit  =  1  percentage  point 


Joint  Strike  Fighter 
Rest  of  portfolio 


Source:  GAO  analysis  of  DOD  data. 
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Significant  Amounts  of  RDT&E  are  Supporting 
Concurrency  or  Upgrade  Efforts  on  Programs  in 
Production 

Program 

Growth  in  last 
year  (millions) 

Reason  for  additional  funding 

Start  of 
production 

Joint  Strike  Fighter 

$3,922 

To  reduce  risk 

2007 

SBIRS  High 

$785 

To  meet  requirements 

2001 

F-22  Raptor 

$780 

For  modernization 

2001 

P-8A  Poseidon 

$742 

For  new  increment  of  capability,  to 
correct  deficiencies,  update  estimates 

2010 

Virginia-class 

$727 

For  enhancements,  cost  reduction 
initiatives,  testing 

1997 

Global  Hawk 

$722 

For  inclusion  of  new  capabilities,  testing 

2001 

DDG51 

$656 

For  inclusion  of  new  capabilities 

1985 

Trident  II 

$624 

For  modernization  and  replacement 

1987 

Apache  Block  IIIA 

$506 

For  software  development 

2010 

Source:  GAO  analysis  of  December  2010  Selected  Acquisition  Reports  and  other  DOD  data. 
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Number  of  MDAPs  Decreased  in  FY  2011 
and  is  Expected  to  Decrease  Further 


•  In  201 1 , 6  programs  estimated  at  $29  billion  entered  the 
portfolio,  4  programs  estimates  at  $108  billion  exited 

Number  of  MDAPs  Fiscal  year  2012  dollars  (in  trillions) 

105  2.0 


•  Looking  forward  to  the  2012  portfolio,  at  least  1  program  is 
expected  to  enter,  13  programs  expected  to  exit 


Source:  GAO  analysis  of  DOD  data. 
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Timing  and  Amount  of  Technology, 
Design,  and  Manufacturing  Knowledge 

Achieved 
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A  Knowledge-Based  Approach  is  Key  to 
Successful  Program  Outcomes 


5  to  6  years  or  less 

Materiel  Develcpment 
Decisicn 

o 

/X 

Development  Start 
/ r\ 

Production  Start 

/X 

Based 

Model 


Technology 

development 


o 

PDR 


t 


Knowledge  Point  1 
Technologies,  time,  funding  and 
other  resources  match  customer  needs. 

Decision  to  invest  in  product  development. 


Integration 


Product  development 

Demonstration 


o 

CDR 

Knowledge  Point  2 
Design  is  stable  and  performs 
as  expected. 

Decision  to  start  building  and  testing 
production  representative  prototypes. 


Production 


t 


Knowledge  Point  3 
Production  meets  cost,  schedule, 
and  quality  targets. 

Decision  to  produce  first  units  for 
customer. 


•  Model  provides  framework  for  incremental,  time  certain  (development  constrained  to  5  to  6  years 
or  less),  and  knowledge-based  approach  to  weapon  system  acquisitions. 

•  Success  requires  structured,  disciplined  application  and  adherence  to  model. 

•  Knowledge  points  align  with  key  investment  inflection  points. 

•  Controls  are  in  place  for  decisions  makers  to  measure  progress  against  specific  criteria  and 
ensure  managers  capture  key  knowledge  before  moving  to  next  phase. 
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Knowledge  At  Three  Critical  Junctures 
Still  Not  Consistent 


•  20  of  37  programs  in  the  current  portfolio  entered 
development  with  critical  technologies  nearing 
maturity;  4  had  technologies  fully  mature 

•  8  of  37  programs  had  stable  designs  at  CDR  or  start 
of  ship  fabrication;  only  5  tested  system-level 
prototypes 

•  26  of  32  programs  plan  to  demonstrate  critical 
processes  on  a  pilot  line  at  production  start;  4  plan 
to  have  these  processes  in  control 

•  15  of  24  programs  plan  to  complete  production 
representative  prototype  testing, 
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Progress  In  Implementing  Acquisition 
Reforms  and  Department  Initiatives 
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New  Acquisition  Reforms  and  DOD  Policy 
Initiatives  Could  Improve  Outcomes 


•  The  Weapon  Systems  Acquisition  Reform  Act  of  2009  inserted 
a  number  of  requirements  whether  programs  planned  to: 

•  Hold  PDR  before  system  development  start. 

•  Complete  competitive  prototyping  as  part  of  technology 
development  phase. 

•  Describe  measures  taken  to  ensure  competition  throughout  the 
program  lifecycle  in  their  acquisition  strategies. 

•  Consider  trade-offs  among  cost,  schedule,  and  performance 
objectives  at  Milestone  B  approval  to  ensure  affordability. 

•  In  addition,  DOD  has  introduced  new  initiatives  intended  to 
control  costs  and  requirements 

•  Early  Materiel  Development  Decision  required  for  all  programs. 

•  Introduction  of  affordability  targets  at  major  program  milestones 

•  Use  of  “should  cost”  to  strengthen  negotiations  with  contractors 
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Programs  Have  Begun  to  Implement 
Reforms  and  New  Initiatives 

•  Programs  in  our  2011  assessment  have  begun  to  implement  acquisition 
reforms  that  could  improve  cost  and  schedule  outcomes. 

•  Early  systems  engineering  -  1 1  of  16  pre-MDAPs  in  our  assessment 
have  scheduled  a  preliminary  design  review  before  Milestone  B. 

•  Competitive  prototyping  -  13  of  16  pre-MDAPs  plan  to  develop 
competitive  prototypes  prior  to  Milestone  B. 

•  Competition  -  1 1  of  16  programs  plan  to  incorporate  competition  nto 
their  acquisition  strategy  after  Milestone  B. 

•  Trade-offs  -  Each  of  the  3  programs  that  entered  system  development 
or  were  re-certified  had  the  requirement  for  making  major  cost, 
schedule,  and  performance  tradeoffs  before  development  start  waived. 

•  Programs  are  still  in  the  process  of  implementing  new  DOD  initiatives. 

•  6  of  16  pre-MDAP  programs  in  our  assessment  reported  holding  MDD. 

•  4  of  the  1 6  future  and  1 9  of  the  37  current  MDAPs  reported  having 
affordability  targets. 

•  6  of  the  1 6  future  and  23  of  the  37  current  MDAPs  reported  having 
“should  costs.” 
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Reforms  and  Initiatives  are  Increasing 
Activities  in  Technology  Development  Phase 


Funding  for  Future  and  Current  Programs  during  Technology  Development 

Fiscal  year  2012  dollars  (in  millions) 


The  focus  on  pre-Milestone  1000+ 
B  activities  results  in 
increased  spending  in  the  750  <1000 
technology  development  500  <750 

phase,  which  could  have 
beneficial  effects  if  the  funds  25o-<5oo 
are  spent  on  activities  such 
as  prototype  demonstrations100'"250 
and  systems  engineering  0  <100 
analysis. 


Percentage  of  future  MDAPs 
Percentage  of  current  MDAPs 
Number  of  programs 


Source:  GAO  analysis  of  DOD  data. 
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Good  trends 

•  Out  with  the  old,  in  with  the  new  -  it’s  improving  the  portfolio’s  health 

•  More  mission  area  reviews  to  reduce  duplication  -  it’s  freeing  funding 

•  More  SE  time  &  energy  from  MDD  through  A  to  B  -  it’s  reducing  risk 

•  CAPEs  new  role  and  responsibility  may  be  best  thing  -  it’s  forcing  change 

Things  to  think  about 

•  MDD  continue  to  force  incremental  solutions  -  it’s  easier  to  plan/execute 

•  Continue  to  scrutinize  reqments  -  it’s  the  only  way  to  reduce  cycle  time 

•  Reconsider  the  role  of  our  S&T  community  -  it  will  impact  EVERYTHING!! 

•  Demand  knowledge  -  it  will  make  things  more  predictable,  less  risky 

•  Demand  a  5-year  cycle  time  from  B  to  IOC  -  it  will  force  knowledge 

•  Find  a  mechanism  to  disseminate/imp  lessons  -  it  creates  role  models 
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RDT&E  Percentage  Cost  Growth  From 
Baseline  per  MDAP 


400% 

350% 

300% 

250% 

200% 

150% 

100% 

50% 

0% 

-50% 


Summary  Analysis 
Average  growth  =  104  percent 
Median  growth  =  32  percent 

Total  Cost  Growth  =  $113  billion 
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Note:  Three  programs  have  greater  than  400  percent  RDT&E  cost  growth,  ranging  from  729%  to  3313%  (GMUjLS, 
MH-60S,  C-130J). 
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Total  Percentage  Cost  Growth  per 
Major  Defense  Acquisition  Program 
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Summary  Analysis 
Average  growth  =  95  percent 
Median  growth  =  36  percent 

Total  cost  growth  =  $447  billion 
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Note:  Two  programs  have  greater  than  600  percent  total  cost  growth  (1365%  and  1523%). 
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Changes  Between  FY  2010  and  FY  2011  MDAP 
Portfolios 


Programs  added  since  FY  2010 

Programs  removed  from  FY  2010 
portfolio 

Total  cost:  $29  billion 

Total  cost:  $108  billion 

Apache  Block  NIB  New  H C/M C- 130 

Build  Recapitalization 

Bradley  Armored  Fighting 
Vehicle  Upgrade 

C-17A  Aircraft 

Small  Diameter  Bomb 
KC-130J  Increment  II 

CVN  68 

EA-6B  Improved  Capability  III 

Minuteman  III  Propulsion 
Replacement  Program 

MC-1 B  Predator  UAS 
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Expected  Changes  Between  FY  201 1  and 
FY  2012  MDAP  Portfolios 


Programs  added  since  FY  2011 

Programs  removed  from  FY  2011  portfolio 

KC-46  Tanker  Replacement 

Program 

Airborne  Signals  Intelligence 
Payload 

Advanced  Threat  Infrared 
Countermeasure/Common 
Missile  Warning  System 

B-2  Radar  Modernization 
Program 

C-5  Avionics  Modernization 
Program 

Expeditionary  Fighting  Vehicle 

F-22  Raptor 

Force  XXI  Battle  Command 
Brigade  and  Below 

Increment  1  Eariy-lnfantry 
Brigade  Combat  Team 

Joint  Mine  Resistant  Ambush 
Protected  vehicle 

Large  Aircraft  Infrared 
Countermeasures 

Longbow  Apache 

Space  Based  Space 
Surveillance  Block  10 

Lewis  and  Clark-class  Dry  Cargo/Ammunition  ship  (T-AKE)  | 
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1yr/5yr/Baseline:  Less  Than  Half  of  MDAPs 
Meet  GAO  High-Risk  Cost-Growth  Targets 


1-year  5-year  Baseline 

comparison  comparison  comparison 

<2%  growth  <10%  growth  <15%  growth 


Programs  that  do  not  meet  criteria 
Programs  that  meet  criteria 


Source:  GAO  analysis  of  DOD  data. 


The  number  of  programs  represents  those  in  the  201 1  portfolio — those  with  December  201 0  SARs — which  break  down  several 

programs  into  smaller  elements  for  reporting  purposes.  One  program,  Airborne  Signals  Intelligence  Payload  (ASIP)  -Baseline,  was  not  24 

included  in  5-year  comparison  because  data  were  not  available  to  make  that  comparison.  BMDS  is  not  included. 


